Lots of little machines can accomplish great things
Parellise your code
Pros
Fault redundency
Forces repeatable/reproducable infrastructure
library(future) makes parallel processing very useable
Cons
Changes to your code for split-map-reduce
Write meta code to handle I/O data and code
Not applicable to some problems
Adopt a split-map-reduce mindset
- Break problems down into stateless lumps
- Reuseable bricks that can be applied to other tasks
Setup a cluster
New in googleComputeEngineR v0.3 - shortcut that launches cluster, checks authentication for you
library(googleComputeEngineR)
vms <- gce_vm_cluster()
#2019-03-29 23:24:54> # Creating cluster with these arguments:template = r-base,dynamic_image = rocker/r-parallel,wait =
#FALSE,predefined_type = n1-standard-1
#2019-03-29 23:25:10> Operation running...
...
#2019-03-29 23:25:25> r-cluster-1 VM running
#2019-03-29 23:25:27> r-cluster-2 VM running
#2019-03-29 23:25:29> r-cluster-3 VM running
...
#2019-03-29 23:25:53> # Testing cluster:
r-cluster-1 ssh working
r-cluster-2 ssh working
r-cluster-3 ssh working
library(future)
googleComputeEngineR has custom method for future::as.cluster
## make a future cluster
library(future)
library(googleComputeEngineR)
vms <- gce_vm_cluster()
plan(cluster, workers = as.cluster(vms))
...do parallel...
Forecasting example
# create cluster
vms <- gce_vm_cluster("r-vm", cluster_size = 3)
plan(cluster, workers = as.cluster(vms))
# get data
my_files <- list.files("myfolder")
my_data <- lapply(my_files, read.csv)
# forecast data in cluster
library(forecast)
cluster_f <- function(my_data, args = 4){
forecast(auto.arima(ts(my_data, frequency = args)))
}
result <- future_lapply(my_data, cluster_f, args = 4)
Multi-layer future loops
Can multi-layer future loops (use each CPU within each VM)
Thanks for Grant McDermott for figuring optimal method (Issue #129)
future_sim <-
## Outer future_lapply() call loops over the no. of VMS
future_lapply(1:length(vms), FUN = function(x) {
## Inner future_lapply() call loops over desired no. of iterations / no. of VMs
future_lapply(1:(iters/length(vms)), FUN = slow_func)
})
CPU utilization
3 VMs, 8 CPUs each = 24 threads
We spoke previously of
Clusters of VMs + Docker = Horizontal scaling
Kubernetes
Clusters of VMs + Docker + Task controller = Kubernetes
Kubernetes
Pros
Auto-scaling, task queues etc.
Scale to billions
Potentially cheaper
May already have cluster in your organisation
Cons
Needs stateless, idempotent workflows
Message broker?
Minimum 3 VMs
Dockerfiles for Shiny apps
Built on Cloud Build upon GitHub push:
FROM rocker/shiny
MAINTAINER Mark Edmondson (r@sunholo.com)
# install R package dependencies
RUN apt-get update && apt-get install -y \
libssl-dev
## Install packages from CRAN needed for your app
RUN install2.r --error \
-r 'http://cran.rstudio.com' \
googleAuthR \
googleAnalyticsR
## assume shiny app is in build folder /shiny
COPY ./shiny/ /srv/shiny-server/myapp/
Kubernetes deployments - Shiny
Shiny App:
kubectl run shiny1 \
--image gcr.io/gcer-public/shiny-googleauthrdemo:latest \
--port 3838
kubectl expose deployment shiny1 \
--target-port=3838 --type=NodePort
Dockerfiles for plumber APIs
Built on Cloud Buid every GitHub push:
FROM trestletech/plumber
# copy your plumbed R script
COPY api.R /api.R
# default is to run the plumbed script
CMD ["api.R"]
Kubernetes deployments - Plumber
R plumber API:
kubectl run my-plumber \
--image gcr.io/your-project/my-plumber \
--port 8000
kubectl expose deployment my-plumber \
--target-port=8000 --type=NodePort
Shiny apps waiting for service
Expose your workloads via Ingress
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: r-ingress-nginx
spec:
rules:
- http:
paths:
- path: /gar/
# app deployed to /gar/shiny/
backend:
serviceName: shiny1
servicePort: 3838
Apps available at URL on demand
curl 'http://mydomain.com/api/echo?msg="its alive!"'
#> "The message is: its alive!"
